Image processing method, image processing apparatus, and electronic camera

ABSTRACT

An image processing method of the present invention includes a detection step detecting a characteristic area from each of three or more shot images having a common graphic pattern in part thereof, the characteristic area having an image significantly different from the other shot images, and a combining step extracting a partial image located in the characteristic area from each of the three or more shot images and combining these partial images into one image.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation based upon and claims the benefit ofpriority from U.S. application Ser. No. 12/068,431 filed on Feb. 6, 2008and Japanese Patent Application No. 2007-034912, filed on Feb. 15, 2007,the entire contents of which is incorporated herein by reference.

BACKGROUND

1. Field

The present invention relates to an image processing method for imagecombining, an image processing apparatus provided with an imagecombining function, and an electronic camera provided with an imagecombining function.

2. Description of the Related Art

Patent reference 1 (Japanese Unexamined Patent Application PublicationNo. 2001-28726) discloses an electronic camera provided with an imagecombining function. The principle of the function is additive averagecombining of multiple shot images obtained by, for example, continuousshooting. Shooting continuously a moving object (dynamic body) using atripod and image-combining the obtained multiple shot images allow astill background and a trajectory of the dynamic body to be included inone image.

It should be noted that this image combining does not discriminate thedynamic body from the background and the background is also included inthe dynamic body area, and thereby the dynamic body appears to betransparent. To perform combining to make a dynamic body opaque(hereinafter, called “opaque combining”), it is necessary to detect apresence area of dynamic body from each of the shot images and toconnect the areas.

However, it is difficult to detect a presence area of dynamic bodyautomatically, and therefore it is currently difficult to realizeautomatic opaque combining.

SUMMARY

The present invention provides an image processing method capable ofperforming opaque combining of shot images without fail. The presentinvention further provides an image processing apparatus and anelectronic camera capable of performing opaque combining of shot imageswithout fail.

An image processing method of the present invention includes a detectingstep detecting a characteristic area from each of three or more shotimages having a common graphic pattern in part thereof, thecharacteristic area having an image significantly different from theother shot images, and a combining step extracting a partial imagelocated in the characteristic area from each of the three or more shotimages and combining these partial images into one image.

Here, the detecting step preferably assumes, in each of the three ormore shot images, an area which has an image significantly differentfrom an averaged image of the other shot images as the characteristicarea.

Also, the detecting step preferably generates a distribution map of thecharacteristic area detected from each of the three or more shot images,and the combining step preferably performs the extraction according tothe distribution map.

Also, the detecting step preferably performs filter processing on thedistribution map for smoothing distribution boundaries.

Also, the combining step may perform weighted average on the partialimage extracted from each of the three or more shot images and partialimages extracted from the same areas in the other shot images.

Also, the combining step may set a weight of the weighted averaging tobe a value specified by a user.

Also, the detecting step preferably performs the detection, instead ofusing the three or more shot images, using reduced-size versionsthereof.

Further, an image processing apparatus of the present invention includesa detecting unit that detects a characteristic area from each of threeor more shot images, the characteristic area having an imagesignificantly different from the other shot images, and a combining unitthat extracts a partial image located in the characteristic area fromeach of the three or more shot images and combines these partial imagesinto one image.

Also, the detecting unit preferably assumes, in each of the three ormore shot images, an area which has an image significantly differentfrom an averaged image of the other shot images as the characteristicarea.

Also, the detecting unit preferably generates a distribution map of thecharacteristic area detected from each of the three or more shot imagesand the combining unit preferably performs the extraction according tothe distribution map.

Also, the detecting unit preferably performs filter processing on thedistribution map for smoothing distribution boundaries.

Also, the combining unit may perform weighted average on the partialimage extracted from each of the three or more shot images and partialimages extracted from the same areas in the other shot images.

Also, the combining unit may set a weight of the weighted averaging tobe a value specified by a user.

Also, the detecting unit preferably performs the detection, instead ofusing the three or more shot images, using reduced-size versionsthereof.

Further, an electronic camera of the present invention includes animaging unit that shoots an object to obtain a shot image and any one ofthe image processing apparatus of the present invention to process threeor more shot images obtained by the imaging unit.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration of an electroniccamera;

FIG. 2 is a diagram illustrating examples of shot images specified by auser;

FIG. 3 is an operational flowchart of a combine-processing part 17A;

FIG. 4 is a diagram illustrating generating steps of an index image (upto difference image calculation);

FIG. 5 is a diagram illustrating generating steps of an index image (upto index image calculation);

FIG. 6 is a diagram illustrating an index image Y_(index);

FIG. 7 is a diagram illustrating a true index image I_(index);

FIG. 8 is a diagram illustrating the true index image I_(index) afterfilter processing;

FIG. 9 is a diagram illustrating a combining method based on the trueindex image I_(index) (in an opacity of 1); and

FIG. 10 is a diagram illustrating a combining method based on the trueindex image I_(index) (in an arbitrary opacity).

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of the present invention will be described.The present embodiment is an embodiment for an electronic camera.

First, a configuration of an electronic camera will be described.

FIG. 1 is a diagram illustrating the configuration of the electroniccamera. As shown in FIG. 1, an electronic camera 10 includes a shootinglens 11, an imaging sensor 12, an A/D converter 13, a signal processingcircuit 14, a timing generator (TG) 15, a buffer memory 16, a CPU 17, animage processing circuit 18, a displaying circuit 22, a rear monitor 23,a card interface (card I/F) 24, an operating button 25, etc., and a cardmemory 24A is attached to the card interface 24. Among these, the CPU 17is capable of performing image-combine processing (to be describedbelow). In the following description, it is assumed that acombine-processing part 17A performing the image-combine processing isincluded in the CPU 17.

The CPU 17 is connected to the buffer memory 16, the image processingcircuit 18, the displaying circuit 22, and the card interface 24 via abus 19. The CPU 17 sends image data to the image processing circuit 18by use of the bus 19 and thereby performs normal image processing on ashot image such as pixel interpolation processing and color conversionprocessing. Also, the CPU 17 sends the image data to the rear monitor 23via the displaying circuit 22, and thereby displays various images onthe rear monitor 23. Further, the CPU 17 writes an image file into thecard memory 24A and reads the image file out of the card memory 24A viathe card interface 24.

Further, the CPU 17 is connected to the timing generator 15 and theoperating button 25. The CPU 17 drives the imaging sensor 12, the A/Dconverter 13, the signal processing circuit 14, etc. via the timinggenerator 15, and also recognizes various indications such as modeswitching provided by a user via the operating button 25. Hereinafter,the electronic camera 10 is assumed to have a shooting mode, an editingmode, and the like.

Next, operation of the CPU 17 in the shooting mode will be described.

When the electronic camera 10 is set to be in the shooting mode, theuser enters a shooting indication into the CPU 17 by manipulating theoperating button 25. The shooting indications include a single shootingindication and a continuous shooting indication, and the CPU 17discriminates the both indications based on a push-down period length ofthe operating button 25, and the like.

When the single shooting indication is entered, the CPU 17 drives theimaging sensor 12, the A/D converter 13 and the signal processingcircuit 14 once, and obtains an image signal (image data) for one frameof a shot image. The obtained image data is stored in the buffer memory16.

At this time, the CPU 17 sends the image data of the shot image storedin the buffer memory 16 to the image processing circuit 18 and performsthe normal image processing on the shot image, and also generates animage file of the processed shot image to write the image file into thecard memory 24A. Thereby the single shooting is completed. Here, theimage file has an image storing area and a tag area, and the image dataof the shot image is written in the image storing area and informationaccompanying the shot image is written in the tag area. The accompanyinginformation includes a reduced-size version of image data of a shotimage (image data of a thumbnail image).

On the other hand, when the continuous shooting indication is entered,the CPU 17 continuously drives the imaging sensor 12, the A/D converter13, and the signal processing circuit 14 multiple times, and obtains animage signal (image data) for multiple frames of shot images. Theobtained image data is stored in the buffer memory 16.

At this time, the CPU 17 sends the image data of the shot images storedin the buffer memory 16 to the image processing circuit 18 and performsthe normal image processing on the shot image, and also generates animage file of the processed shot image to write the image file into thecard memory 24A. After this operation has been performed for multipleframes of the shot images, the continuous shooting is completed. Here,each of the image files has an image storing area and a tag area, andthe image data of the shot image is written in the image storing areaand information accompanying the shot image is written in the tag area.The accompanying information includes a reduced-size version of imagedata of a shot image (image data of a thumbnail image).

Next, operation of the CPU 17 in the editing mode will be described.

When the electronic camera 10 is set to be in the editing mode, the CPU17 displays a menu of the editing mode on the rear monitor 23. One ofthe menu items is “image combining”. This image combining is an imagecombining capable of the opaque combining.

While the menu is displayed, the user manipulates the operating button25 to specify a desired item of the menu to the CPU 17. When the userspecifies “image combining”, the CPU 17 reads out the image files in thecard memory 24A, and sends the image data of the thumbnail images addedto the image files to the displaying circuit 22. Thereby, shot imagespreviously obtained are reproduced and displayed on the rear monitor 23.In this reproducing display, a plurality of shot images is preferablydisplayed in parallel at the same time on the rear monitor 23 so thatthe user can compare a plurality of the shot images to one another.

While the shot images are reproduced and displayed, the user manipulatesthe operating button 25 to specify to the CPU 17 desired N shot imagesI_(k) (k=1, 2, . . . , N) among the shot images reproduced anddisplayed. Then, the user manipulates the operating button 25 to specifyan opacity α_(k) (k=1, 2, . . . , N) of a dynamic body included in eachof the specified N shot images I_(k) (k=1, 2, . . . , N). The opacityα_(k) is an opacity of a dynamic body M_(k) included in the shot imageI_(k).

Here, the N shot images I_(k) (k=1, 2, . . . , N) specified by the userhave a part common to one another (background) except for a dynamicbodies M_(k) (k=1, 2, . . . , N) as shown in FIG. 2. Such shot imagesI_(k) (k=1, 2, . . . , N) are obtained under common shooting conditions(shooting sensitivity, shutter speed, aperture value, and framing).

In the present embodiment, these N shot images I_(k) (k=1, 2, . . . , N)need not include a shot image in which the dynamic body M_(k) is notpresent, but, instead, the number of shot images N is required to bethree or more.

Also, pixel coordinates of the dynamic bodies M_(k) (k=1, 2, . . . , N)included in each of the shot images I_(k) (k=1, 2, . . . , N) preferablydo not overlap with one another. This is because image combiningcalculation in the present embodiment assumes that pixel coordinates dono overlap.

Also, a range of an opacity α_(k) (k=1, 2, . . . , N) which a user canspecify is 0≦α_(k)≦1. For example, for the opaque combining, the useronly needs to specify the opacity α_(k) as α₁=α₂=α₃= . . . =α_(N)=1, andfor erasing all the dynamic bodies from a combined image, specify theopacity α_(k) as α₁=α₂=α₃= . . . =α_(N)=0.

When the opacity α_(k) is specified as above, the CPU 17 sends imagedata of the specified N shot images I_(k) (k=1, 2, . . . , N) to thecombine-processing part 17A and performs image-combine processing on theshot images. The CPU 17 obtains one combined image by this image-combineprocessing, and then displays the combined image on the rear monitor 23.The CPU 17 also newly generates an image file of the combined image andwrites the image file into the memory card 24A.

Next, operation of the combine-processing part 17A will be described indetail.

For simplicity of the description, specified shot images are assumed tobe three shot images I₁, I₂, and I₃ shown in FIG. 2. In this case, thenumber of shot images N is three. Also, each shot image I_(k) is assumedto have a Y component, a Cb component, and a Cr component.

Accordingly, in the description, the Y component of a shot image I_(k)is denoted by Y_(k), the Cb component of a shot image I_(k) is denotedby Cb_(k), and the Cr component of a shot image I_(k) is denoted byCr_(k). Also in the description, a pixel value of an arbitrary image Xat a pixel coordinates (i,j) is denoted by X(i,j) and the origin of thepixel coordinates (i,j) is determined to be at upper left corner of animage.

FIG. 3 is an operational flowchart of the combine-processing part 17A.Each step thereof will be described in sequence as follows.

(Steps S1 to S3)

The combine-processing part 17A performs size reduction processing oneach of the shot images I₁, I₂, and I₃ in order to improve processingspeed of the image-combine processing. For achieving a size reductionratio of 16, the size reduction processing is performed using thefollowing formulas, for example.

$\begin{matrix}{{{Y_{k}( {i,j} )} = {( {\sum\limits_{y = 4}^{{4\; j} + 3}{\sum\limits_{{jx} = {4\; i}}^{{4\; i} + 3}{Y_{k}( {x,y} )}}} )/16}}{{{Cb}_{k}( {i,j} )} = {( {\sum\limits_{y = 4}^{{4\; j} + 3}{\sum\limits_{{jx} = {4\; i}}^{{4\; i} + 3}{{Cb}_{k}( {x,y} )}}} )/16}}{{{Cr}_{k}( {i,j} )} = {( {\sum\limits_{y = 4}^{{4\; j} + 3}{\sum\limits_{{jx} = {4\; i}}^{{4\; i} + 3}{{Cr}_{k}( {x,y} )}}} )/16}}} & ( {{Formula}\mspace{14mu} 1} )\end{matrix}$

(Step S4)

The combine-processing part 17A generates an average image I_(ave) ofthe shot images I₁, I₂, and I₃ after the size reduction processing.Y_(ave): the Y component of the average image I_(ave), Cb_(ave): the Cbcomponent of the average image I_(ave), and Cr_(ave): the Cr componentof the average image I_(ave), are calculated by the following formulas.

$\begin{matrix}{{{Y_{ave}( {i,j} )} = {( {\sum\limits_{k = 1}^{N}{Y_{k}( {i,j} )}} )/N}}{{{Cb}_{ave}( {i,j} )} = {( {\sum\limits_{k = 1}^{N}{{Cb}_{k}( {i,j} )}} )/N}}{{Cr}_{ave}( {i,j} )} = {( {\sum\limits_{k = 1}^{N}{{Cr}_{k}( {i,j} )}} )/N}} & ( {{Formula}\mspace{14mu} 2} )\end{matrix}$

(Step S5)

The combine-processing part 17A generates an index image Y_(index) ofthe Y component, an index image Cb_(index) of the Cb component, and anindex image Cr_(index) of the Cr component as provisional index images,respectively.

Representing these index images, a generation step of the index imageY_(index) will be described.

First, the combine-processing part 17A focuses on the shot image I₁ asshown in FIG. 4, and calculates a difference image ΔY₁ between the Ycomponent image Y₁ of the focused shot image I₁ and an average image ofthe Y components Y₂ and Y₃ of the other shot images I₂ and I₃.

Similarly, the combine-processing part 17A focuses on the shot image I₂,and calculates a difference image ΔY₂ between the Y component image Y₂of the focused shot image I₂ and an average image of the Y components Y₁and Y₃ of the other shot images I₁ and I₃.

Similarly, the combine-processing part 17A focuses on the shot image I₃,and calculates a difference image ΔY₃ between the Y component image Y₃of the focused shot image I₃ and an average image of the Y components Y₁and Y₂ of the other shot images I₁ and I₂.

Thereby, the difference image ΔY₁ regarding the shot image I₁, thedifference image ΔY₂ regarding the shot image I₂, and the differenceimage ΔY₃ regarding the shot image I₃ are obtained. Here, the differenceimages ΔY₁, ΔY₂, and ΔY₃ are calculated using the following formula.

ΔY _(k)(i,j)=|(Y _(ave)(i,j)×N−Y _(k)(i,j)/(N−1)−Y _(k)(i,j)|  (Formula3)

In this formula, “k” is an image number of a focused shot image, and(Y_(ave)×N−Y_(k))/(N−1) is the average image of the other shot images.

Subsequently, the combine-processing part 17A compares the differenceimages ΔY₁, ΔY₂, and ΔY₃ for every pixel coordinates as shown in FIG. 5.

Then, the combine-processing part 17A determines an area as acharacteristic area A₁, the area being in the difference image ΔY₁ andwhose pixel value is larger than the pixel values of the same areas inthe other difference images ΔY₂, and ΔY₃. This characteristic area A₁ isan area having an outstanding pixel value in the shot image I₁ comparedwith the other shot images I₂ and I₃. Therefore, this characteristicarea A₁ can be assumed to be a presence area of a dynamic body M₁.

Also, the combine-processing part 17A determines an area as acharacteristic area A₂, the area being in the difference image ΔY₂ andwhose pixel value is larger than the pixel values of the same areas inthe other difference images ΔY₁, and ΔY₃. This characteristic area A₂ isan area having an outstanding pixel value in the shot image I₂ comparedwith the other shot images I₁ and I₃. Therefore, this characteristicarea A₂ can be assumed to be a presence area of a dynamic body M₂.

Also, the combine-processing part 17A determines an area as acharacteristic area A₃, the area being in the difference image ΔY₃ andwhose pixel value is larger than the pixel values of the same areas inthe other difference images ΔY₁, and ΔY₂. This characteristic area A₃ isan area having an outstanding pixel value in the shot image I₃ comparedwith the other shot images I₁ and I₂. Therefore, this characteristicarea A₃ can be assumed to be a presence area of a dynamic body M₃.

Then, the combine-processing part 17A generates one index imageY_(index) as a distribution map of these characteristic areas A₁, A₂,and A₃.

The characteristic area A₁ in this index image Y_(index) is providedwith a pixel value “1” as same as the image number of the shot image I₁and the dynamic body M₁, the characteristic area A₂ in the index imageY_(index) is provided with a pixel value “2” as same as the image numberof the shot image I₂ and the dynamic body M₂, and the characteristicarea A₃ in the index image Y_(index) is provided with a pixel value “3”as same as the image number of the shot image I₃ and the dynamic bodyM₃.

Such an index image Y_(index) is calculated by use of the followingformula.

Y _(index)(i,j)=k·of·max[ΔY ₁(i,j), . . . ,ΔY _(N)(i,j)]  (Formula 4)

In this formula, k·of·max[x₁, x₂, . . . , x_(N)] is an element number kof the largest element x_(k) among N elements x₁, x₂, . . . , x_(N).

FIG. 6 is a diagram illustrating the index image Y_(index). As shown inFIG. 6, in the index image Y_(index), many pixels located in thepresence area of the dynamic body M₁ (refer to FIG. 2) have a pixelvalue “1”, many pixels located in the presence area of the dynamic bodyM₂ (refer to FIG. 2) have a pixel value “2”, and many pixels located inthe presence area of the dynamic body M₃ (refer to FIG. 2) have a pixelvalue “3”. Meanwhile, in the other areas in the index image Y_(index), apixel having a pixel value “1”, a pixel having a pixel value “2”, and apixel having a pixel value “3” are mixed.

Therefore, a distribution relationship among the dynamic bodies M₁, M₂,and M₃ (refer to FIG. 2) is reflected to the index image Y_(index) witha certain accuracy. Using this index image Y_(index) enables the dynamicbodies M₁, M₂, and M₃ to be extracted from the shot images I₁, I₂, andI₃, respectively.

However, in this index image Y_(index) there exists an indefiniteportion such as a portion enclosed by a dotted line in FIG. 6. Thereason is probably that the luminance of a part of the dynamic body M₃(refer to FIG. 2) is close to the luminance of the background and falsedetection occurred so that this part was determined to be thecharacteristic area A₁ or the characteristics area A₂, rather thandetermined to be the characteristic area A₃.

Accordingly, the combine-processing part 17A in the present stepperforms the following processing when detecting the characteristicareas A₁, A₂, and A₃ in the calculation of the index image Y_(index)(refer to FIG. 5).

That is, the combine-processing part 17A compares pixel values in thecharacteristic area A₁ of the difference image ΔY_(I), thecharacteristic area A₂ of the difference image ΔY₂, and thecharacteristic area A₃ of the difference image ΔY₃ with a thresholdvalue thY, and assumes an area having pixel value smaller than thethreshold value thY to be a particular area A₀ which does not belong toany of the characteristic areas A₁, A₂, and A₃. Then, thecombine-processing part 17A assigns the particular area A₀ of the indeximage Y_(index) with a particular value except for “1”, “2”, and “3”(hereinafter, “0”). This particular value is replaced with anappropriate value in the next step.

In the present step, the index image Cb_(index) of the Cb component andthe index image Cr_(index) of the Cr component are also generated, andthis is performed for the purpose of this replacement.

Here, the index image Cb_(index) is calculated by use of the followingformulas.

ΔCb _(k)(i,j)=|(Cb _(ave)(i,j)×N−Cb _(k)(i,j))/(N−1)−Cb _(k)(i,j)|

Cb _(index)(i,j)=k·of·max[ΔCb ₁(i,j), . . . ,ΔCb _(N)(i,j)]  (Formula 5)

Also, when calculating this index image Cb_(index), thecombine-processing part 17A performs the following processing.

That is, the combine-processing part 17A compares pixel values in acharacteristic area A₁ of a difference image ΔCb₁, a characteristic areaA₂ of a difference image ΔCb₂, and a characteristic area A₃ of adifference image ΔCb₃ with a threshold value thCb, and assumes an areahaving a pixel value smaller than the threshold value thCb to be aparticular area A₀ which does not belong to any of the characteristicareas A₁, A₂, and A₃. Then the combine-processing part 17A assigns theparticular area A₀ in the index image Cb_(index) with a particular valueexcept for “1”, “2”, and “3” (hereinafter, “0”).

Also, the index image Cr_(index) is calculated by use of the followingformulas.

ΔCr _(k)(i,j)=|(Cr _(ave)(i,j)×N−Cr _(k)(i,j))/(N−1)−Cr _(k)(i,j)|

Cr _(index)(i,j)=k·of·max[ΔCr ₁(i,j), . . . ,ΔCr _(N)(i,j)]  (Formula 6)

Also, when calculating this index image Cr_(index), thecombine-processing part 17A performs the following processing.

That is, the combine-processing part 17A compares pixel values in acharacteristic area A₁ of a difference image ΔCr₁, a characteristic areaA₂ of a difference image ΔCr₂, and a characteristic area A₃ of adifference image ΔCr₃ with a threshold value thCr, and assumes an areahaving a pixel value smaller than the threshold value thCr to be aparticular area A₀ which does not belong to any of the characteristicareas A₁, A₂, and A₃. Then the combine-processing part 17A assigns theparticular area A₀ in the index image Cr_(index) with a particular valueexcept for “1”, “2”, and “3” (hereinafter, “0”).

(Step S6)

The combine processing part 17A determines whether a pixel having theparticular value “0” exists in the index image Y_(index), and, if itexists, replaces the pixel value with a pixel value at the same pixelcoordinates in the index image Cb_(index).

Further, the combine-processing part 17A determines whether a pixelhaving the particular value “0” remains in the index image Y_(index)after the replacement, and if it remains, replaces the pixel value witha pixel value at the same coordinates in the index image Cr_(index). Theindex image Y_(index) after these replacements is determined to be atrue index image I_(Index) (refer to FIG. 7).

Accordingly, in the present step, indefiniteness of the index imageY_(index) is compensated by the other index images Cb_(index) andCr_(index), and the more definite true index image I_(index) isobtained.

Therefore, even if the luminance of a part of a dynamic body M_(k)included in a certain shot image I_(k) is close to the luminance of abackground thereof, the true index image I_(index) is compensated tobecome more definite as long as the color of the part is different fromthe color of the background.

Here, even in the true index image I_(index) after the compensation,there still remains a possibility that a pixel having a particular value“0” exists. This is because there can exist a part of the dynamic bodywhich is similar to the background thereof in both luminance and color.

(Step S7)

The combine-processing part 17A performs filter processing on the trueindex image I_(index). A filter used at this time is a filter smoothingboundary lines of the characteristic areas A1, A2, and A3, and, forexample, a majority filter of 3×3 pixels. The majority filter has afunction to replace a pixel value of a focused pixel with a mode valueof pixel values of peripheral pixels thereof.

Here, when the mode value becomes the particular value “0” during thisfilter processing, the combine-processing part 17A replaces the pixelvalue of the focused pixel with a second mode value, not the mode value.This replacement eliminates a pixel having the particular value “0” fromthe true index image I_(index).

The filter processing on the true index image I_(index) may be performedonce, but preferably performed appropriate number of times of two ormore. As the filter processing is repeated, pixels having the same pixelvalue (each of the characteristic areas A₁, A₂, and A₃) on the trueindex image I_(Index) gather together gradually. Therefore, in the trueindex image I_(index) after the filter processing, the characteristicarea A₁ covers the entire presence area of the dynamic body M₁ (refer toFIG. 2), the characteristic area A₂ covers the entire presence area ofthe dynamic body M₂ (refer to FIG. 2), and the characteristic area A₃covers the entire presence area of the dynamic body M₃ (refer to FIG.2), as shown in FIG. 8.

In such a true index image I_(index), boundary lines between thecharacteristic areas A₁, A₂, and A₃ are not always located at a boundaryarea between the background and the dynamic body M₁, a boundary areabetween the background and the dynamic body M₂, or a boundary areabetween the background and the dynamic body M₃, but located definitelyin a boundary area between the dynamic body M₁ and the dynamic body M₂,a boundary area between the dynamic body M₂ and the dynamic body M₃, anda boundary area between the dynamic body M₁ and the dynamic body M₃.

Here, in the present step, instead of increase in the number ofexecutions of the filter processing, a filter diameter for the filterprocessing may be increased.

(Step S8)

The combine-processing part 17A performs size enlargement processing onthe true index image I_(index) after the filter processing, and makes asize thereof as same as that of the original shot images I₁, I₂, and I₃.If the size reduction ratio in the steps S1 to S3 described hereinaboveis 16, the size enlargement processing uses, for example, the followingformulas, where a true index image after the enlargement processing isdenoted by I′_(index). In fact, the size enlargement processing usingthe following formulas is a padding processing.

$\begin{matrix}{{{I_{index}^{\prime}( {{4\; j},{4\; i}} )} = {I_{index}( {i,j} )}}{{I_{index}^{\prime}( {{4\; j},{{4\; i} + 1}} )} = {I_{index}( {i,j} )}}{{I_{index}^{\prime}( {{4\; j},{{4\; i} + 2}} )} = {I_{index}( {i,j} )}}{{I_{index}^{\prime}( {{4\; j},{{4\; i} + 3}} )} = {I_{index}( {i,j} )}}{{I_{index}^{\prime}( {{{4\; j} + 1},{4\; i}} )} = {I_{index}( {i,j} )}}{{I_{index}^{\prime}( {{{4\; j} + 1},{{4\; i} + 1}} )} = {I_{index}( {i,j} )}}\vdots {{I_{index}^{\prime}( {{{4\; j} + 3},{{4\; i} + 3}} )} = {I_{index}( {i,j} )}}} & ( {{Formula}\mspace{14mu} 7} )\end{matrix}$

Here, the true index image after the enlargement processing I′_(index)is used in the next step as a source map for extracting a partial imagefrom each of the shot images I₁, I₂, and I₃.

For this purpose, a pixel value in the true index image I′_(index)should be any one of “1”, “2”, and “3” and should not be an intermediatevalue among “1”, “2”, and “3”. Therefore, in the size enlargementprocessing in the present step, an average interpolation should not beapplied, and, if interpolation is applied, it should be a majorityinterpolation.

(Step S9)

The combine-processing part 17A, on the basis of the true index imageI′_(index), extracts a partial image I₁·A₁ located in the characteristicarea A₁ from the shot image I₁, a partial image I₂·A₂ located in thecharacteristic area A₂ from the shot image I₂, and a partial image I₃·A₃located in the characteristic area A₃ from the shot image I₃, as shownin FIG. 9. Then, the combine-processing part 17A combines the threeextracted partial images I₁-A₁, I₂·A₂, and I₃·A₃ to obtain one combinedimage I. Here, a partial image located in an area A of an image X isrepresented by X·A.

At this time, the combine-processing part 17A sets an opacity of thepartial image I₁·A₁ an opacity of the partial image I₂·A₂, and anopacity of the partial image I₃·A₃ to be α₁, α₂, and α₃, respectively.These values α₁, α₂, and α₃ are the opacities specified by the user forthe dynamic bodies M₁, M₂, and M₃ (refer to FIG. 2). This concept isillustrated as shown in FIG. 10.

That is, regarding the characteristic area A₁, weighted averaging isperformed on the shot image I₁ and an average image (I₂+I₃)/2 of theother shot images I₂ and I₃. A weight ratio of the weighted averaging isset to be α₁:(1−α₁) according to the user's specification.

Focusing on this characteristic area A₁, there exists not only thedynamic body M₁ but also the background portion in the shot image I₁,but the graphic pattern of the background portion is as same as that ofthe average image {(I₂+I₃)/2} of the other shot images I₂ and I₃.Therefore, a graphic pattern of the characteristic area A₁ in thecombined image I becomes the background on which the dynamic body M₁ issuperimposed with an opacity of α₁.

Also, regarding the characteristic area A₂, weighted averaging isperformed on the shot image I₂ and an average image (I₁+I₃)/2 of theother shot images I₁ and I₃. A weight ratio of the weighted averaging isset to be α₂:(1−α₂) according to the user's specification.

Focusing on this characteristic area A₂, there exists not only thedynamic body M₂ but also the background portion in the shot image I₂,but the graphic pattern of the background portion is as same as that inthe average image {(I₁+I₃)/2} of the other shot images I₁ and I₃.Therefore, a graphic pattern of the characteristic area A₂ in thecombined image I becomes the background on which the dynamic body M₂ issuperimposed on with an opacity of α₂.

Also, regarding the characteristic area A₃, weighted averaging isperformed on the shot image I₃ and an average image (I₁+I₂)/2 of theother shot images I₁ and I₂. A weight ratio of the weighted averaging isset to be α₃:(1−α₃) according to the user's specification.

Focusing on this characteristic area A₃, there exists not only thedynamic body M₃ but also the background portion in the shot image I₃,but the graphic pattern of the background portion is as same as that inthe average image {(I₁+I₂)/2} of the other shot images I₁ and I₂.Therefore, a graphic pattern of the characteristic area A₃ in thecombined image I becomes the background on which the dynamic body M₃ issuperimposed with an opacity of α₃.

Here, the combining of the shot images I₁, I₂, and I₃ in the presentstep is performed for each component of the shot images I₁, I₂, and I₃,and the common true index image I′_(index) is used for the combining ofeach component. When a Y component of the combined image I is denoted byY, a Cb component of the combined image I is denoted by Cb, and a Crcomponent of the combined image I is denoted by Cr, combining eachcomponent uses the following formulas.

k(i,j)=I′ _(index)(i,j)

Y(i,j)=α_(k(i,j)) ×Y _(k(i,j))(i,j)+(1−α_(k(i j)))×(Y _(ave)(i,j)×N−Y_(k(i,j))(i,j))/(N−1)

Cb(i,j)=α_(k(i,j)) ×Cb _(k(i,j))(i,j)+(1−α_(k(i,j)))×(Cb_(ave)(i,j)×N−Cb _(k(i,j))(i,j))/(N−1)

Cr(i,j)=α_(k(i,j)) ×Cr _(k(i,j))(i,j)+(1−α_(k(i,j)))×(Cr_(ave)(i,j)×N−Cr _(k(i,j))(i,j))/(N−1)  (Formula 8)

That is, a pixel value Y (i,j) at a certain pixel coordinates (i,j) ofthe combined image I is determined to be a weighted-average of a pixelvalue Y₁ (i,j), Y₂ (i,j), and Y₃ (i,j) at the same pixel coordinates inthe shot images I₁, I₂, and I₃ according to a pixel value I′_(index)(i,j) at the same pixel coordinates (i,j) in the true index imageI′_(index). For example, when the pixel value I′_(index) (i,j) of thetrue index image I′_(index) is “1”, a weighted average of the pixelvalue Y₁ (i,j) and an average value of the Y₂ (i,j) and Y₃ (i,j) iscalculated with a weight ratio of α₁:(1−α₁). Also, for example, when thepixel value I′_(index) (i, j) of the true index image I′_(index) is “2”,a weighted average of the pixel value Y₂ (i,j) and an average value ofthe Y₁ (i,j) and Y₃ (i,j) is calculated with a weight ratio ofα₂:(1−α₂).

Similarly, a pixel value Cb(i,j) of a certain pixel of the combinedimage I is determined to be a weighted-average of a pixel value Cb₁(i,j), Cb₂ (i,j), and Cb₃ (i,j) at the same pixel coordinates in theshot images I₁, I₂, and I₃ according to a pixel value I′_(index) (i,j)at the same pixel coordinates (i,j) in the true index image I′_(index).For example, when the pixel value I′_(index) (i,j) of the true indeximage I′_(index) is “1”, a weighted average of the pixel value Cb₁ (i,j)and an average value of the Cb₂ (i,j) and Cb₃ (i,j) is calculated with aweight ratio of α₁:(1−α₁). Also, for example, when the pixel valueI′_(index) (i,j) of the true index image I′_(index) is “2”, a weightedaverage of the pixel value Cb₂ (i,j) and an average value of the Cb₁ (i,j) and Cb₃ (i,j) is calculated with a weight ratio of α₂:(1−α₂).

Similarly, a pixel value Cr(i,j) of a certain pixel of the combinedimage I is determined to be a weighted-average of a pixel value Cr₁(i,j), Cr₂ (i,j), and Cr₃ (i,j) at the same pixel coordinates in theshot images I₁, I₂, and I₃ according to a pixel value I′_(index) (i,j)at the same pixel coordinates (i,j) in the true index image I′_(index).For example, when the pixel value I′_(index) (i,j) of the true indeximage I′_(index) is “1”, a weighted average of the pixel value Cr₁ (i,j)and an average value of the Cr₂ (i,j) and Cr₃ (i,j) is calculated with aweight ratio of α₁:(1−α₁). Also, for example, when the pixel valueI′_(index) (i,j) of the true index image I′_(index) is “2”, a weightedaverage of the pixel value Cr₂ (i,j) and an average value of the Cr₁(i,j) and Cr₃ (i,j) is calculated with a weight ratio of α₂:(1−α₂) (Thatis the description of the step S9.).

As a result, the combine-processing part 17A, while managing opacitiesof the dynamic bodies M1, M2, and M3 included in the shot images I₁, I₂,and I₃ to be α₁, α₂, and α₃ specified by the user, respectively, cancombine these shot images I₁, I₂, and I₃ correctly. When the opacitiesare set to be “1”, the combining may become the opaque combining.

Here, while the user in the present embodiment specified the opacitiesof all the dynamic bodies, the opacities of some or all of the dynamicbodies may not be specified. In this case, the CPU 17 sets the opacitywhich is not specified to be a default value (e.g., “1”).

Also, although the combine-processing part 17A in the present embodimenteliminated the particular value “0” from the true index image I_(index)during the filtering processing (step S7), the processing associatedwith this elimination may be omitted. Note that, in this case, thecombine-processing part 17A needs to replace the particular value “0”with a non-particular value (any one of “1”, “2”, and “3”) in the end ofthe step S7.

For example, the combine-processing part 17A replaces the particularvalue “0” with “1” when the shot image I₁ takes priority, the particularvalue “0” with “2” when the shot image I₂ takes priority, and theparticular value “0” with “3” when the shot image I₃ takes priority.Here, which of the shot images I₁, I₂, and I₃ takes priority may bespecified by the user in advance or determined automatically by thecombine-processing part 17A through its evaluation of the shot imagesI₁, I₂, and I₃.

Also, while the combine-processing part 17A in the present embodimentperformed the size reduction processing (steps S1 to S3) on the shotimages to improve the processing speed in the image-combine processing,the size reduction processing (steps S1 to S3) may be omitted and,instead, the thumbnail image of the shot image (stored in the same imagefile as the shot image) may be used.

Also, while the combine-processing part 17A in the present embodimentperformed the size reduction processing (steps S1 to S3) on the shotimages to improve the processing speed in the image-combine processing,the size reduction processing (steps S1 to S3) may be omitted when aslow processing speed does not matter. In this case, the sizeenlargement processing (step S8) is also omitted.

Although, in the electronic camera 10 of the present embodiment, thewhole image-combine processing was performed by the CPU 17, a part orwhole of the image-combine processing may be performed by a dedicatedcircuit except for the CPU 17 or the image processing circuit 18.

Also, some or all of the functions of the image-combine processing inthe electronic camera 10 of the present embodiment may be provided inthe other apparatus having a user interface and a monitor such as animage storage or a printer.

Also, some or all of the functions of the image-combine processing inthe electronic camera 10 of the present embodiment may be performed by acomputer. When performed by a computer, a program for the purpose(image-combine processing program) is stored in a memory of the computer(hard disk drive or the like). Install of the image-combine processingprogram into the hard disk drive is performed, for example, via theInternet or a recording medium such as a CD-ROM.

The many features and advantages of the embodiments are apparent fromthe detailed specification and, thus, it is intended by the appendedclaims to cover all such features and advantages of the embodiments thatfall within the true spirit and scope thereof. Further, since numerousmodifications and changes will readily occur to those skilled in theart, it is not desired to limit the inventive embodiments to the exactconstruction and operation illustrated and described, and accordinglyall suitable modifications and equivalents may be resorted to, fallingwithin the scope thereof.

1. An image processing method, comprising: a detecting step detecting acharacteristic area from each of three or more shot images having acommon graphic pattern in part thereof, said characteristic area havingan image significantly different from the other shot images; and acombining step extracting a partial image located in said characteristicarea from each of said three or more shot images and combining thesepartial images into one image.
 2. The image processing method accordingto claim 1, wherein said detecting step assumes, in each of said threeor more shot images, an area which has an image significantly differentfrom an averaged image of the other shot images as said characteristicarea.
 3. The image processing method according to claim 1, wherein: saiddetecting step generates a distribution map of said characteristic areadetected from each of said three or more shot images; and said combiningstep performs said extracting according to said distribution map.
 4. Theimage processing method according to claim 3, wherein said detectingstep performs filter processing on said distribution map for smoothingdistribution boundaries.
 5. The image processing method according toclaim 1, wherein said combining step performs weighted average on thepartial image extracted from each of said three or more shot images andpartial images extracted from the same areas in the other shot images.6. The image processing method according to claim 5, wherein saidcombining step sets a weight of said weighted averaging to be a valuespecified by a user.
 7. The image processing method according to claim1, wherein said detecting step performs said detecting, instead of usingsaid three or more shot images, using reduced-size versions thereof. 8.An image processing apparatus, comprising: a detecting unit detecting acharacteristic area from each of three or more shot images, saidcharacteristic area having an image significantly different from theother shot images; and a combining unit extracting a partial imagelocated in said characteristic area from each of said three or more shotimages and combines these partial images into one image.
 9. The imageprocessing apparatus according to claim 8, wherein said detecting unitassumes, in each of said three or more shot images, an area which has animage significantly different from an averaged image of the other shotimages as said characteristic area.
 10. The image processing apparatusaccording to claim 8, wherein: said detecting unit generates adistribution map of said characteristic area detected from each of saidthree or more shot images; and said combining unit performs saidextracting according to said distribution map.
 11. The image processingapparatus according to claim 10, wherein said detecting unit performsfilter processing on said distribution map for smoothing distributionboundaries.
 12. The image processing apparatus according to claim 8,wherein said combining unit performs weighted average on the partialimage extracted from each of said three or more shot images and partialimages extracted from the same areas in the other shot images.
 13. Theimage processing apparatus according to claim 12, wherein said combiningunit sets a weight of said weighted averaging to be a value specified bya user.
 14. The image processing apparatus according to claim 8, whereinsaid detecting unit performs said detecting, instead of using said threeor more shot images, using reduced-size versions thereof.
 15. Anelectronic camera, comprising: an imaging unit that shoots an object toobtain a shot image; and an image processing apparatus according toclaim 8 that processes three or more shot images obtained by saidimaging unit.